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ABSTRACT OF THE DISCLOSURE 

Although negative example can t>e highly useful to better 
understand the user's' needs In content-t)ased image retrieval, It was 
considered by few authors. We address some issues related to the combination 
of positive and negative examples to perform a more efficient image retrieval. A 
relevance feedback approach that uses positive example to perform 
generalization and negative example to perform specialization is described 
herein. In this approach, a query containing both positive and negative example 
Is processed in two steps. The first step considers positive example only in 
order to reduces the set of images participating In retrieval to a more 
homogeneous subset Then, the second step considers t>oth positive and 
negative examples and. , acts on the Images retained in the first step. 
Mathematically, relevance feedback is formulated as an optimization of Intra 
and inter variances of positive and negative examples. 
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TITLE OF THE INVENTION 

Content-Based Image Retrieval Using Positive and Negative 

Examples 

FIELD OF THE INVENTION 

[0001] The present invention relates to Image retrieval* More 

specifically, the present invention is concerned with content-based image 
retrieval using positive and negative examples. 

BACKGROUND OF THE INVENTION 

[0002] With advances in the computer technologies and the advent 

of the Worid-Wide Web, there has been an explosion in the quantity and 
complexity of digital data being generated, stored, tfansmitted* analyzed, and 
accessed [30]. These data take different forms such as text, sound, images and 
videos. 

[0003] The huge number of images available renders it necessary to 

develop systems for efRcient image retrieval which can help users locate the 
needed images in a reasonable time. These systems use many attributes of the 
images, such as the presence of a particular combination of colors or the 
depiction of a particular type of event [19]. Such attributes may either be 
derived from the content of the image or from its surrounding text and data. 
This leads to various approaches in image retrieval such as content-based 
techniques and text-based techniques. A good survey on the techniques used 
in image retrieval can be found in [2], [19] and [29]. 
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[0004] If we focus on content-based inrtage retrieval, we can 

distinguish many ways of fomiulating queries. Early systems such as QBIC [18] 
asl< the user to select image features such as color, shape, or texture. Other 
systems like BLOBWORLD [4] require the user to provide a weighted 
combination of features. However, it is generally difficult to directly specify the 
features needed for a particular query, for several reasons. First, because not 
all users understand the image vocabulary (e.g. contrast, texture, color) needed 
to fomiulate a given query. Second, even if the user is an image specialist, it is 
not easy to translate the images the user has in mind into a combination of 
features. 

[0005] An alternative approach is to allow the user to specify the 

features and their corresponding weights implicitly via a visual interface known 
as "query by example". The user can choose Images that will participate in the 
query and weight them according to their resemblance to the images sought. 
The results of the query can then be refined repeatedly by specifying more 
relevant images. This process, called Relevance Feedback (RF), is defined in 
[28] as the process of automatically adjusting an existing query using 
information fed back by the user about the relevance of previously retrieved 
documents. Relevance feedback is used to model the user subjectivity in 
several stages. First, it can be applied to identify the ideal images that are in 
the user's mind. At each step of the retrieval, the user is asked to select a set of 
images which will participate in the query; and to assign a degree of relevance 
to each of them. This information can be used in many ways in order to define 
an analytical fomi representing the query intended by the user. The ideal query 
can be defined independently from previous queries, as in [25]. It can also 
depend on the previous queries, as in the query point movement method where 
the ideal query point is moved towards positive example and away from 
negative example [8]. Examples of this method can be found in [28] and [20]. 
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The computation of the ideal query may be unnecessary in certain cases such 
as the current work, where we automatically integrate what the user is looking 
for into the similarity measures without the need to identify any ideal point 

[0006] Relevance feedback allows also to capture the user's needs 

by assigning a degree of importance (e.g* weight) to each feature or by 
transforming the original feature space into a new one that best corresponds to 
the user's needs and specificities. This is done by enhancing the importance of 
those features that help in retrieving relevant images and reducing the 
importance of those which do not Once the importance of each feature is 
determined, the results are applied to define similarity measures which 
correspond better to the similarity intended by the user in the current query. 

[0007] The operation of attributing weights to features can also be 

applied to perform feature selection, which is defined In [31] as the process of 
choosing a subset of features by eliminating redundant features or those 
providing little or no predictive information, in fact, after the importance of each 
feature is determined, feature selection can be performed by retaining only 
those features which are important enough; the rest will be eliminated. By 
eliminating some features, retrieval performance can be improved because in a 
low-dimension feature space, it is easier to define good similarity measures, to 
perform retrieval in a reasonable time, and to apply effective indexing 
techniques [19]. 

[0008] While many studies have focused on how to learn from user 

interaction in relevance feedback [26, 30, 25, 23, 3, 9. 17], few of them evoked 
the relevance of negative example [21, 7, 20, 5]. Negative example can be 
useful for query refinement since it allows to determine the images the user 
doesn't want in order to discard them. In [7], the authors show that when they 
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use only positive feedbacic. the major improvement occurs at the first feedback 
step; and that the improvement with positive and negative feedbacit is 
remaritable for the four first steps where the results continuously get better. We 
also note that relevance feedback with negative example is useful to reduce 
noise (undesired images that have been retrieved) and to decrease the miss 
(desired images that have not been retrieved). 

[0009] Relevance feedback using positive example has been 

considered by many authors [1. 30. 26, 25. 24, 9, 17]. Ishikawa et al. 1251 
define a quadratfe distance function for comparing images. Consider that we 
have a query consisting of N images, each image is represented by an I- 
dimension feature vector =[x,„...,jc^f, where T denotes matrix 
transpositton. Consider also that the user associates each image participating 
in the query with a degree of relevance which represents its degree of 
resemblance with the sought images. The authors compute two paiametera. 
namely the ideal query q = igu^,.,q,f and the ellipsoid distance matrix W, that 
nrdnimi^ the quantity D given in Equation (1), which represents the global 
distance between the query images and the ideal query. 

N 

[0010] In [26], Rui. et al. decompose each image into a set of I 

features, each of which is a vector of reals. Let be flie 1* feature vector of 
the n* queiy image and fr„ ttie degree of relevance assigned by the user to the 
n* Image. Assume also that the query consists of N images. The authore 
compute, for each feature i, the ideal query vector g,, a matrix Wi and scalar 
weight ut which minimize the global dispersion of the query images given by 
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Equation (2). By minimizing the dispersion of tlie query images, they try to 
enhance the concentrated features. i.e., features for which example images are 
dose to each other. 

t=l n=l (2) 

[0011] In [27], the authors propose to use the same model but with 

negative degrees of relevance assigned to negative example images. This 
consideration leads to neglect the relevant features of negative example, so 
that negative example will be confused with positive example. 

[0012] If we consider the way negative example was interpreted in 

previous studies, we can distinguish two categories of models. In the first 
category, the positive example images are selected by the user; however, the 
negative example images are chosen automatically by the retrieval system 
among those not selected by the user. In the second category, both positive 
and negative example images are chosen by the user. 

[0013] We will begin by examining the first category, which includes 

the work of MQiler et al. [7]. Concerning the Initial query, they propose to enrich 
it by automatically supplying non-selected images as negative example. For 
refinement, they select the top 20 images resulting from the previous query as 
positive feedback. As negative feedback, they choose four of the non-returned 
images. Their system performs refinement through several feedback steps; in 
each step, it tries to move the ideal query towards the positive example and 
away from the negative example. This is done by using the Rocchio formula 
[12]: 
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^1 ^ ^2 ^ (3) 

where Q is the Ideal query, ni and n2 are the numbere of positive and negative 
images in the query respectively, and Ri and S| are the features of the positive 
and negative images respectively, a and p detemiine the relative weighting of 
the positive and negative examples. They adopt the values a ^ 0.65 and p = 
0.35 used by some text-retrieval systems [7]. We notice that in this category of 
models, since the system selects negative example images automatically, it Is 
very Important to use the right images, othenvise this can destroy the query, 
indeed, if tlie system chooses as negative example some images which should 
rether be considered as positive example, then the relevant features of these 
images will be discarded, and this will mislead the retrieval process. 



[0014] The second category includes the work of Vasconcelos and 

Lippman [20] where they propose a Bayesian model for image retrieval, 
operating on the assumption that the database is constituted of many image 
classes. When performing retrieval, they support image classes that assign a 
high memberehip probability to positive example images, and penalize image 
classes that assign a high memberehip probability to negative example images. 
We notice that the authore consider that the positive and the negative 
examples have the same relative importance. 

[0015] In [23, 24], Picard et al. organize the datebase Images Into 

many hierarchical trees according to individual features such as color and 
texture. When the user submits a query, they perform comparison using each 
of the trees, then they combine the resulting sete by choosing those image 
sete which most efRciently describe positive example, with the condition that 
these sete dont describe negative example well. 
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[0016] In [21], Beikin et al. use a Bayesian probabilistic model in 

which they assume that the relevant features of positive example are good, 
whether or not they are relevant to negative example. Their interpretation of 
negative example is that the context In which positive example appears Is 
inappropriate to the searcher's problem. They propose to increase the 
(positive) weight of the relevant features of positive example (irrespective of 
their appearance in negative example): and to enhance (with negative weights) 
the relevant features of negative example which dont appear in positive 
example. Enhancing important f^res of positive example which also appear 
in negative example, can mislead the retrieval process, as will be discussed 
hereinbelow. 

[00171 In [6, 5]. the authors consider that the Image database Is 

made up of relevant images, among which the user chooses positive example, 
and non-relevant images, among which the user chooses negative example. 
They use a probabilistic model In which they try to estimate the distribution of 
relevant images and to simultaneously minimize the probability of retrieving 
non-relevant Images. 

OBJECTS O F THE INVENTIOM 

10018] An object of the present invention is therefore to provide 

improved content-based image retrieval using positive and negative examples. 

[0019] other ol^jects. advantages and features of the present 

Invention will become more apparent upon reading of the following non- 
restrictive description of prefened embodiments thereof, given by way of 
example only with reference to the accompanying drawings. 
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[0020] it is to be noted that various documents are referred to 

herein. These docunrtents are hereby incorporated by reference in their 
entirety. 

BRIEF DESCRIPTIOM OF THE DRAWINGS 
[0021] In the appended drawings: 

[0022] Figure 1 illustrates precision-scope ounces for two cases: 

negative example in two steps and negative example in one step; 

[0023] Figure 2 illustrates a positive average; a negative average, 

and the overall query average; 

[0024] Figure 3 illustrates the minimization of the global dispersion 

leading to neglect the relevant features of negative example; 

[0025] Figure 4 illustrates the minimization of the dispersion of 

positive example, the minimization of negative example and the minimization of 
the distinction between them; 

[0026] Figure 5 illustrates an example of the images a user has to 

choose from; 

[0027] Figure 6 illustrates a series of choice the user may make 

regarding any of the seen pictures; 

[0028] Figure 7 illustrates the decomposition of the HIS color space 

into a set of subspaces and the computation of each subspace's histogram; 
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[0029] Figure illustrates and example of the results of a search done 

with positive example only; 

[0030] Figure 9 illustrates an example of reHnement with both 

positive and negative examples; 

[0031] Figure 10 illustrates precision-scope curves for retrieval with 

positive example and refinement with negative example; and 

[0032] Figure 11 Is a table on the nunfri[)er of iterations needed to 

locate a given category of images in two cases: using positive example only 
and using bottt positive and negative examines. 

DESCRIPTION OF THE EMBODiMEMT 

[0033] When an image retrieval system returns the results of a given 

query, we often encounter two problems: noise and miss. Noise consists of 
retrieved images which dont conespond to what the user wants. Miss Is the set 
of images conespondbig to what the user wants which have not been retrieved. 
These two problems (noise and miss) occur because of imperiections at 
different levels. For the user, it may not be easy to formulate an adequate 
queiy using the available images, either because none of them correspond to 
what the user wants or because the user lacte knowledge of imagery details 
such as the meaning of features. For the retrieval system, it may be difficult to 
translate the user's needs and speclfidtles In terms of image featores and 
similarity measures. However, the use of negative example in query fomiuiation 
and refinement can help to reduce noise and miss. Indeed, after the rasuite of a 
given query are obtained, the user can maintain the positive example images 
and enrich the query by including some undesired images as negative 
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example. This implies that images similar to those of negative example will be 
discarded, thus reducing noise. At the same time, the discarded images will be 
replaced by others which would have to resemble better what the user wants. 
Hence, the miss will also be decreased. Furthermore, the user can find, among 
the recently retrieved images, more images that resemble what the user needs 
and use them to formulate a new query. Thus, the use of negative example 
would help to resolve what is called the page zero problem, i.e., that of finding 
a good query image to initiate retrieval. By mitigating the page zero problem, 
the retrieval time will be reduced and the accuracy of the results will be 
improved [19]. We also note that relevance feedback with negative example is 
useful when, in response to a user feed-back query, the system retums exactly 
the same images as in a previous iteration. Assuming that the user has already 
given the system all the possible positive feedback, the only way to escape 
from this situation is to choose some images as negative feedback [20]. 

[0034] Now, let us examine how negative example can be 

interpreted and how this could be applied to relevance feedback. Existing 
systems consider negative example in two ways. 

[0035] Some systems like [23] consider it at image level. They 

search for the set of images similar to positive example, then they search for 
the set of images similar to negative example; and finally they manipulate the 
two sets in order to obtain the set of images that they return to the user. 

[0036] Other systems such as the one in [21] consider the negative 

example at feature level. They try to identify and enhance the features which 
help to retrieve images that are at the same time similar 1o positive example but 
not similar to negative example. 
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[0037] in what follows, we will consider the negative example from 

the feature point of view, and use it to identify the most discriminating features 
according to the user-given query. 

fP038] In knowledge discovery in databases, most leaming-from- 

example algorithms partition the set of examples into positive and negative 
subsets. They perform generalization using positive example and specialization 
using negative example [22]. These algorithms try to extract decision rules 
including characteristic rules and discrimination rules* A characteristic mie of a 
set is an assertion which characterizes a concept satisfied by all or most of the 
members of this set For example, the symptoms of a specific disease can be 
summarized by a characteristic rule. A discrimination rule is an assertion which 
discriminates a concept of ttie target set from the rest of the database [10]. For 
example, to distinguish one disease from others, a discrimination rule should 
summarize the symptoms that discriminate this disease from others. 

[0039] Before we see how to apply the above-described principle to 

image retrieval, let us make the following hypothesis. We always assume that 
positive and negative examples possess some relevant features that are 
discriminant, i.e., relevant to either positive or negative example or to both but 
whose values are not the same in positive and in negative examples. In other 
words, we exclude the case in which the relevant features of positive example 
are the same as those of riegative example, with similar values, in such a case, 
the query is ambiguous. The system rejects it and ask the user to specify a new 
one. 

[0040] Now, a first idea to apply the above-described principle to 

image retrieval would be the following. First, characteristic rules can be 
extracted from positive example images by the identification of their relevant 
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features. More importance must be given to such features fn the retrieval 
process and images enhancing them should be retrieved. Second, 
discrimination rules can be extracted from the difference between positive 
example and negative example. Relevant features whose values are not 
common to positive and negative exampies are good discriminators, and hence 
must be given more importance; conversely, common features are not good 
discriminators, and must be penalized. However, by applying this principle in 
this manner, we may mislead the retrieval process by neglecting certain 
relevant features of positive and negative examples, as explained below. 

[0041] To make this idea clearer, let us first define what we mean by 

a relevant feature. We consider that a given feature is relevant if it helps to 
retrieve the images being sought. In our case, this will depend on two lactors. 

[0042] First, the relevance can be considered with respect to the 

query. A feature relevant to the query is a feature which is salient in the 
majority of the query images. As we hereinbelow, we will consider every feature 
whose values are concentrated in the query images, and which discriminates 
well between positive and negative examples, as relevant to the query. 

[0043] Second, the relevance of a feature can be considered with 

respect to the database. If a given feature's values are almost the same for the 
majority of the database images, then this feature is not relevant since it 
doesn't allow to distinguish the sought images from the others; and vice versa. 
To illustrate, consider a database in which each image contains an object with 
a circular shape but the color of the object difl^ers from one image to another. In 
such a database, the feature Shape is not interesting for retrieval since It 
doesn't allow to distinguish between desired and undesired Images; however, 
the feature Color is interesting. In other words, a feature in terni of which the 



CA 02397424 2002-08-09 



13 

database is homogeneous is not relevant for retrieval; whereas, a feature In 
term of which the database is heterogeneous is relevant. 

[0044] Now. let us analyze the consequences of neglecting features 

whose values are common to both positive and negative examples. In fact, this 
depends on the nature of the database. If the database is homogeneous in 
terms of such features, then neglecting them will not be disastrous since they 
are not relevant to the database. On the other hand, if the database is 
heterogeneous in terms of these features, then neglecting them will lead the 
system to retrieve many undesired images and to miss many desired images. 

[0045] To develop a solution that works for any query, It is clear that 

common features should be considered. However, in some cases, there are not 
enough common features to be considered alone at a given moment; they must 
rather be considered together with other features. 

[0046] . According to an aspect of the present invention, a retrieval In 

two steps is proposed. A first step serves to reduce the heterogeneity of the 
set of images participating in the retrieval by restricting It to a more 
homogeneous subset according to positive example relevant features (and thus 
according to common features also). In this first step, we enhance all the 
relevant features of positive example. We rank the database images according 
to their resemblance to positive example and then retain only the Nbi top- 
ranked images, where Nbi is a predetemnined number 

[0047] Only images retained in the first step will participate In the 

refinement performed in a second step, where we enhance the discrimination 
features, i.e., those whose values are not common to positive and negative 
examples, in this second step we rank the candidate Images according to their 
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similarity to positive example and dissimilarity to negative example, and return 
to the user only the Nb2 (Nb2 < Nbi) top-ranked images. IHence, even if the 
common features are neglected in the second step, this will not mislead the 
retrieval since they were considered in the first step. We confirmed 
experimentally, using the retrieval system described hereinbelow, the 
importance of processing queries with negative example in two steps. 

[0048] Figure 1 compares the curves precision-scope for the two 

techniques: negative example queries processed in two steps (the adopted 
technique) versus negative example queries processed in a unique step (in 
which both positive and negative examples are considered and all images in 
tiie database participate in retrieval). Precision is the average of relevance of 
retrieved images, and scope is the number of retrieved images. It is clear from 
Figure 1 that when queries containing negative example are considered in one 
step, the precision of retrieval decreases quid^ly with the number of retrieved 
images. 

[0049] Some special cases are important and merit to be mentioned 

to show that the proposed method functions as well. These cases emerge 
when all the discrimination features come from positive example only or from 
negative example only. Indeed, if the relevant features of positive example are 
strictly included in fliose of negative example and with common values, then 
applying the proposed principle leads, in the first step, to enhance the relevant 
features of positive example (which are the same as the common features) and 
to retain images looking like it. Then, in the second step, to enhance the rest of 
the negative example relevant features and to discard images near to it. On the 
other hand, if the relevant features of negative example are strictly included in 
those of positive example and with common values, then applying the proposed 
principle leads, in the first step, to enhance the relevant features positive 
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example (which include those of negative example) and to retain images 
looking like the positive example. Then, in the second step, to enhance only 
those features relevant to positive but not to negative example and to re-rank 
the images according to these features essentially. 

[0050] A last question that should be answered is: can the user 

compose a query using negative example only? 

piOSI] First we note that, for a given query, the number of non- 

relevant Images is usually much higher than the number of relevant images. In 
other words, if we know what someone doesn*t want, this doesnt Inform us 
sufficiently about what the user wants. For example, if the user gives an Image 
of a car as negative example without giving any positive example, then we 
cannot know whether the user is looking for images of buildings, animals, 
persons or other things. Nevertheless, we think that negative example can be 
used alone in some cases, for Instance, to eliminate a subset from a database. 
Suppose that the database contains, in addition to images the user agrees 
with, other images that the user's culture doesn't tolerate, e.g. nudity images for 
some persons. In such a case, the user can first eliminate the undesired 
images by using some of them as negative example; then the user can 
navigate in, or retrieve from the rest of the database. Concerning the retrieval 
system, the negative-example-only query will be considered as a positive 
example query, i.e., the system first searches for images that resemble 
negative example. Then, when the resulting images (images that the user 
wants to discard) are retrieved, the system retums to the user the rest of the 
database rather these images. 

[0052] As explained hereinabove, the present invention generally 

aims at defining a retrieval scenario where the user can select positive example 
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images, negative example images, and their respective degrees of relevance. 
This allows first to reduce the heterogeneity of the dataset on the basis of the 
positive example, then to refine the results on the basis of the negative 
example. 

[0053] Via the interaction with the user, we globally aim to achieve 

two objectives. First, to be able to combine the query images together with their 
respective degrees of relevance in order to identify what the user is loolcing for, 
and to integrate this information in similarity measures. Second, to weight each 
feature and its components according to Its relevance to the query and the 
discrimination power it can provide. Before we give details on our formulation of 
relevance feedback, we will begin by explaining the adopted image model and 
similarity measure. To represent images, we have adopted a hierarchical model 
like the one in [26] where each image, either in the query or In the database, is 
represented by a set of I features, each of which is a real vector of many 
components. This choice ensures a good modeling of both images and image 
features, and a reduction in the computation time [26]. The hierarchical two- 
level image model implies the need to choose a distance metric for each level. 
For feature level, we have chosen a generalized Euclidean distance function, 
as in [25]. If Xf^and Xi^sre the i^ feature vectors of the images Xi and X2 
respectively, then the distance at this feature level is 

where Wi is a symmetric matrix that allows us to define the generalized 
ellipsoid distance D|. 

[0054] The choice of this distance metric will allow us not only to 

weight each feature's component but also to transform the initial feature space 
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Into a space that better models the user's needs and specificities [26]. The 
global distance between two images xi and xa is linear and is given in Equation 

D{xi, X2) = ^Uiixii - X2i)^Wi{aSii - X2i) 

fei (4) 

where uj is the global weight assigned to the i^ feature. 

[0055] Now that the image model and similarity metrics have been 
chosen, let us explain how to compute the parameters Ui and W| which fulfill the 
objectives set in the previous section. Consider that the user constructs a query 
composed of Ni positive example images and their respective relevance 
degrees ttI for n - 1 Ni, as well as N2 negative example images and their 

respective relevance degrees n:^ for n = 1,...,N2. (It should be noted that ;r^is 
not the square of ^„ ; 2 is an index designating the negative example). In the 
first step, we consider only the positive example. We want to enhance each 
feature and its components according to their relevance to the positive 
example. This can be done by introducing ttie optimal parameters ui and Wi 
which minimizB Jposnive, the global dispersion of positive example, given In 
Equation (5). 



JposiUve = 53 ^ 52 - ^if^Wi{£ii - xl) 

i=l n=:l (5) 

where xj is the weighted average of positive example (Rgure 2), given by 
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[0056] The problem formulation and the computation of optimal 

parameters are the same as in [26] excepted that the full set of query images is 
replaced by the positive example images only. The basic Idea is to give more 
weight to features and feature components for which the positive example 
images are close to each other in the feature space* An informal justification is 
that if the variance of query images is high along a given axis, any value on this 
axis is apparently acceptable to the user, and therefore this axis should be 
given a low weight, and vice versa [25]. 



[0057] in the second step, we consider both positive and negative 

example images, and the refinement concerns the images retained in the first 
step. Let us first define Jgiobai. the global dispersion of the query, including 
positive and negative example images: 

/ 2 iVfc 

1=1 Af=ln«l (7) 

where Ic = 1 for positive example and k = 2 for negative example, and where , 

given in Equation (8), is the weighted average of all query images for the i*^ 
feature. See Figure 4. 

jLfk^l Z^n=l (8) 

[0058] Rui et al. proposed in [27] to allocate negative degrees of 

relevance to negative example images and to compute the parameters which 
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minimize the same expression of Equation (7). Let us analyze the 
consequences of such an approach. If we separate positive example from 
negative example in Equation (7), then we can write 

[0059] They choose > 0 for n » 1,...,Ni and < 0 for n = 1,...,N2 

. We thus obtain: 

/ ATi J Nit 

(0) 

[0060] where | \ designates the absolute value of tvI . Equation 

(9) shows that the global dispersion Jgtobai is nothing but the dispersion of 
positive example minus the dispersion of negative example. Hence, by 
minimizing the global dispersion, even if the authors move the global query 
average q (with which they compare their images) towards positive example 
and away from negative example, two problems emerge. 

[0061] First, minimizing the global dispersion will lead to minimize 

the dispersion of positive example, but with respect to the global query average 
q rather than the positive example average . This will not give an optimal 
minimization of the positive example dispersion; and hence, the relevant 
features of positive example will not be given enough importance. 



[0062] Second, minimizing the global dispersion will lead to 

maximize the dispersion of negative example. This implies that they neglect the 
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relevant features of negative example. Hence, their retrieval system will not be 
able to discard the undesired images. See Figure 3 for an illustration. 



[0063] Before describing our fomnulation of the problem in detail, let 

us recall the general objectives. By introducing the weights U| and Wu we want 
to give more importance to the relevant features of either positive or negative 
example which allow to distinguish well between them. In other words, via U{ 
and Wi we want to attribute weights to features and to transform our feature 
space into a new space in which positive example images are as close as 
possible, negative example images are as close as possible, and positive 
example is as far as possible from negative example. (See Figure 4). To 
translate our objectives into a mathematical fomnulation, we start by 
distinguishing positive example images from negative example images in flie 
global dispersion formula of Equation (7). For each feature i, we recall the 
weighted average of positive example images ^/and we define the weighted 

average of negative example images ffin Equations (10) and (11) 
respectively. 



(10) 

[0064] By introducing x/and into Equation (7), we can rewrite it 

as follows: 
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(12) 

[006^ Developing Equation (12) £^ves 

= [ ( i: E - - 1?)) + ( i: e «5(«it< - - «)) 

2 iVfc 2 /Vfc 1 

+( E E - QifWiislu - ^?)) ( E E - ^fwiii^ - gi)) 

(13) 

[0066] It can easily be shown that the second and third parts of 

Equation (13) are zero. For example, the second part 

ELl 'Enil <(^ni - SiVWii^, - fi) = ELi [( E^X - ^iV) Wii^ - ft)] 

= ELi [((ESi^^i) - 0:^1 ^)^) V,(^ - = 0 

since, according to Equations (10) and (11), 

[0067] Thus, Equation (1 3) can be written as follov\^: 



JsuM = z E - sfm(x** - 5f )]+ «)^wi(^ - «) = a^r 

(14) 

[0068] The first term A expresses the positive example internal 

dispersion, i.e., how close positive example images are to each other, added to 
the negative example internal dispersion, i.e., how close negative example 
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images are to each other. The secx>nd term R expresses the distance between 
the two sets, i.e., how far positive example is from negative example. 

[0069] By distinguishing the intra dispersion A from the inter 

dispersion R, it is now clearer how we can formulate our objectives in a 
mathematical problem. In fact, we want to compute the model parameters, 
namely U| and Wi, which minimize the Intra dispersion A and maximize the Inter 
dispersion R. Several combinations of A and R are possible. We have chosen 

to compute the parameters which minimize the ratio — , assuming that i! 9^0. 

In the case of J{sO, the positive example and the negative example are not 
distinguishable and the query is ambiguous. In such case, the query is lejected 
and the user is asked to fbmnulate a new one. Furthennore, to avoid numerical 

stability problems, we introduce the following two constraints: Y' — ^land 

det(Wi)=1 for ail 1=1,..., I. By using Lagrange multipliers, the optimal parameters 
ui and W| must minimize the quantity L given in Equation (15). 

^ = I - ^(E^ - l) - EA*(de«(Wi) - 1) 

where 

»=1 A=l n=l 

and 

^ = E«^ E **(^ - m'^Wii^ - 5) 

»=1 ft=l 



(15) 



(16) 



(17) 
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^'denotes the sum of positive example relevance degrees, i.e.. ^' «y^',ff! 
and ^'denotes the sum of negative example relevance degrees, l.e., 

^070] The optimization problem in oreler to obtain the optimal 

parameters U| and Wj will now be resolved. 

[0071] Before starting, we note that the relative importance of 

positive and negative examples must be chosen, i.e., ^' vwth respect to 
Some image retrieval systems [7] adopt the values used by certain text retrieval 
systems which are 0.65 for positive example and 0.35 for negative esoimple. 
Other systems such as the one in [20] assume that positive example and 
negative example have the same importance. We have adopted for the latter 
choice because it allows some simplifications in the derivation of the problem. 
FurUiermore, we normalize all the user-given relevance degrees so that 

E0072] To obtain the optimal solution for W|, we take the partial 

derivative of L witti respect to w^^for r.ss1 H|, where l-i| is the dimension of 

the i* feature and is the re*" element of W|, i.e.. W, = [w^^ ] . We have 

dwi^. ^ir. (18) 



where 



flA 2 N,, 

*" *=i»=» (19) 
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and 

ft=l (20) 

[0073] Before computing - — , we note that 

det(fr|) = 2jj;,(-l)'*'w,^ dBt(W^^), where det(fF^_)i8 the re** minor of W| oistained 
by eliminating the r"* row and the s"* column of det(^,) . Hence. 



dwi,^ (21) 
By 6ut>8tituting Equations (19), (20) and (21) in (18), we obtain 

dL 



(22) 

[0074] Now consider the matrix W^'^ » [w^^] , the inverse matrix of Wi 

(provided that Wi Is invertible). To obtain the value of each component , we 
use the determinant method for matrix inversion to obtain 
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Knowing that Glet(Wi)=1 , we find 

[0075] In Equation (23), we replace det(^^) by its value from 

Equation (22) to ot>tain 

, r 2 iV* 2 -I 

' L *=ln=l fc=i J 

(24) 

where 

[0076] Equation (24) can also be written in matrix form as 

(25) 

where Q is the matrix [c^^] such that 

2 iVjb 2 
A:=rlns::l jb=l 

(26) 

[0077] Now we will compute the value of y independently from A 

which Is an unknown parameter. We can write Equation (25) as follows: 
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-L 

but since detiwr') = 1 , then y = (det(C,))''' Q"^ . Finally, the optimal solution for 
W| is given by Equation (27) 

Wi = 7c?r' = (rfe*(a))*cr' ^27) 

where the components of Ci are given by Equation (26). 

[0078] Let us analyze, roughly, the effect of the dispersion of 

positive and negative examples on the components of Wi. First, we can write 
Equation (26) in a matrix fonri. as follows: 

d — RCovoi — ACoWi 

where Covai is the sum of intra oovarlance matrices for the i*'' feature, i.e.. 
Cava, = [cov ] SMdn that 

and Covn Is the inter covariance matrix for the i* feature, i.e., Covr, =[co\r,Ji 
such that 

[0079] Now, consider Equation (28), where we set the values of A 

and R since they concern all the features. If the intra dispersion is high relative 
to the inter dispersion, and hence the elements of Covai are important relative 
to the elements of Covn then, according to Equation (28). the values of the 
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components of Ci will be Important. But since = yCg~' (Equatlon(27)). it follows 

that the values of >t^,^wlll be small; and consequently, the 1^ feature's 

components will be given low weights. On the other hand, if the intra dispersion 
Is low relative to the Inter dispersion for the 1^ feature, by a similar line of 
reasoning, we see that this feature's components will be given high weights. 
This behavior of Wi fulfiils our objective of enhancing discriminant features 
against other ones. 

[0080] To obtain the optimal solution for ui, we talce the partial 

derivative of L with respect to ui. 



(29) 



where 



dA ^ ^ 



^ ib=ln=l (30) 



and 



^ *=i (31) 
[0081] By substituting Equations (30) and (31 ) in (29), we otitain 

(32) 

[0082] We multiply both sides of Equation (32) by ui. to obtain: 
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^ (33) 



where 



2 Nk 2 

(34) 

[0083] Now, to get rid of the unknown parameter A , we will try to 

find a relation, Independent of A, between ui and any uj. First A can be 
computed directly from Equation (33) as follows: 



^ (35) 
10084] Second, taking the sum on i of EquaHon (33) gives 

H^Ujfj+JiR'^^^O, but since Z.^-'U then = 0. It 



foWom that 



B? (36) 
[0085] Equations (35) and (36) imply that for every feature i 



i=i (37) 
[0086] It follows from Equation (37) that fiu^ = f^ul « ... » f,uf » f^j . 



10087] 



Hence, 
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V J3 (38) 

[0088] Finally, to obtain the optimal solution of ui, we replace Uj in 

Equation (37) by its value from Equation (38). We find the following: 

y/Ji (39) 

[0089] The optimal solution for ui is given by Equation (39), where fi 

is defined tiy Equation (34). 

[0090] Let us analyze, roughly, the influence of the dispersion of 

positive and negative examples on the value of each ui. First, we can write fi in 
Equation (34) as 



fi = RFai - AFn 



where 

2 Nk 



and 



ft=ln=l (41 J 



ft=i (42) 
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[0091] We assume that A and R have constant values since they 

depend on all the features, if, for the feature, the intra dispersion is high 
relative to the inter dispersion, then the quantity Fai will gain In importance 
relative to the quantity Fn. According to Equation (40), this will increase the 
value of fi. Moreover, Equation (39) shows that when fi increases, ui decreases; 
and hence, the i^ feature will be given a low weight. Conversely, If, for the i^ 
feature, the intra dispersion is low relative to the inter dispersion, then, by a 
similar line of reasoning, we find that the feature will be given a high weight. 
Therefore, the optimal value that we found for Ui fulfills our objective of 
enhancing the relevant discriminant features against others. 

[0092] The input to the algorithm consists of positive example 

images, negative example images and their respective relevance degrees. The 
output consists of the optimal parameters W| and uj. These parameters are 
computed according to Equations (27) and (39), respectively. The computation 
of these parameters requires the computation of f/, q,, fi, A and R 
according to Equations (10), (11), (8). (34). (16) and (17), respectively. Our 
algorithm is iterative because the computation of W| and ui depends on A and 
R, and the computation of A and R depends on Wi and U]. We use the fixed 
point method to peribrm the computation of W| and U|. An initialization step is 
required, in which we adopt the following values: 



[0093] 



We initialize Wi with the diagonal matrix 
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where 



(Tit 



1 



2 Nh 
A=ln=l 



is the standard deviation of the component of the i"^ feature computed for the 
full set of query images. 



[0094] 

by 



We initiaiize the parameter ui with a Idnd of dispersion given 



where 



ELi E&i - ^i)'^Wi(si, - g?) 



[OOSq The computation of W| requires the inversion of the matrix Q. 

l-iowever. in the case of (Ni-i-N2)<H|, Q is not invertilMe. in [25], the authors 
suggest proceeding by singuiar vaiue decomposition (SVO) to obtain the 
pseudo inverse matrix. This solution doesnt give a satisfactory resuit. 
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especially when (Ni+N2)is far less than Hi [26]. The authors of [26] propose. In 
the case of a singular matrix, to replace Wi by a diagonal matrix whose 

elements are the inverse of the standard deviation, i.e., hv if r = s and 

=^0 elsewhere. In our case, we propose to replace Wi by a diagonal matrix 
whose elements are the inverse of the diagonal elements of the matrix Q, i.e.. 



mil 



where w;^ = — and can be obtained by setting r = s in Equation (26). 



Example of application 



[0096] The relevance feedback model was validated in the context 

of image retrieval. To do so, we implemented a content-based image retrieval 
system In which we present the user virith a graphical Interface displaying nine 
sample images related to different subjects and emphasizing different features 
(see Figure 5). The user can choose more Images from the database before 
formulating the query. To add an image to the query, the user clicks on its 
"Select" button. The system displays a dialog box allowing the user to specify a 
degree of relevance (see Figure 6). 



[0097] The jsossible relevance degrees are 
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• Very similar corresponds to the relevance value 2 for a 
positive example image. 

• Similar: corresponds to the relevance value 1 for a positive 
example image. 

• Doesnt matter the image will not participate in the query. 

• Different: corresponds to the relevance value 1 for a negative 
example Image. 

• Very different: corresponds to the relevance value 2 for a 
negative example image. 

[0098] Once the query formulation is completed, the user can launch 

the retrieval process by clicking on the "Begin search" button. The system 
computes the optimal parameters and performs comparison in three ways 
depending on the constitution of the query. If the query contains only positive 
example images, then the optimal parameters are those which minimize 
Equation (5). and the database images are raniced In increasing order 
according to their distance from the positive example average. This distance is 
given by Equation (43). We return the top-ranlced images to the user. 

/ 

^1 (43) 

[0099] If the query contains only negative example images, then the 

system proceeds initially by a similar procedure, but considering the negative 
example rather than the positive example. This means that the system 
computes the ideal parameters which minimize the dispersion of negative 
example images, ranl<s the images in increasing order according to their 
distance from the negative example average, then returns to the user the last- 



en, 02397424 2002-08-09 



34 



ranked images. If the query contains both positive and negative examples, then 
the system perfomis the two steps of retrieval. The parameter computation and 
the distance function used in the first step are the same as in the case of a 
positive-example-only query. The database images are ranked in increasing 
order and only the Nbi first ones are retained for the second step. In the 
second step, ttie system computes the parameters which minimize Equation 
(15), then the retained images are ranked acconjing to their closeness to the 
positive example and their farness from the negative example. The comparison 
function is given by Equation (44). Finally, the system returns the Nb2 top- 
ranked images to the user. 

(44) 

[00100] Each image, either in the database or in the query, is 

represented by a set of 27 feature vectors, computed as follows: First, we map 
every pixel in the image to a point in the 3-D HSI space (Figure 7). This 
operation consists of computing, for every triple [H,S.I], the number of pixels 
having the values Hue = i-l, Saturation = S and Intensity = I. We obtain a 3-D 
color histogram that we can't use, as it takes up a lot of space and most of its 
values are zeros. For example, if we have an image with HSI values ranging 
between 0 and 255, our histogram will contain 256^ cells, most of which dont 
correspond to any pixel. 

[00101] To reduce the histogram^s size, many solutions are possible, 

such as the spatial repartition of the points of the 3-D histogram, taking into 
account their respective occurrence frequency, i.e., the number of pixels 
corresponding to each point in the histogram. However, as the main aim of our 
work is not finding the best visual features, but rather developing a relevance 
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feedback technique, we have adopted a simple solution which consists of 
partitioning the space by subdividing the axes H, S and I into three equal 
intervals each. This gives us 3^ = 27 subspaces, as shown in Figure 7. Each 
subspace will constitute a feature, and we compute its con^sponding vector as 
follows. The subspace is subdivided into 2^ = 8 sub-subspaces. We compute 
the sum of the elements of each sub-subspace and we put the result in the 
corresponding cell of the feature vector (Figure 7). We note that our feature 
estimation method is somewhat similar to the quantizer function used in [15]. 

Experimental results and performance evaluation 

Example of refinement with negative example 

[00102] Tests were performed on 10.000 images from The 

Pennsylvania State University [16, 13]. This database contains Images related 
to different subjects, emphasizing different features, and taken under different 
illumination conditions. For each image, the set of features is computed as 
explained above. We perfomned many tests for retrieval and refinement. Even 
when positive and negative examples are not readily distinguishable, a system 
according to the present invention succeeded in identifying discrimination 
features and sorting the resulting images according to these features. Figure 8 
gives an example of retrieval with positive example only. Figure 9 gives and 
example of retrieval with positive and negative examples. 

[00103] In the first example, two images participated in the query as 

positive example. Both of these images contain a green tree under the blue sky 
(5095.ppm and 5118.ppm). Figure 8 shows the top nine retumed images. We 
notice that the two query images are retumed in the top positions. There are 
also some other images containing trees under the sky, but we notice the 
presence of noise consisting of three images of a brown bird on a green tree 
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under the blue sky (5523.ppm, 5522.ppm, 5521. ppm). At the same time, there 
have been miss, because the database contains other images of trees under 
the si^ that have not been retrieved, in the second example, we try to refine 
the results of the first example. IHence, we use the same images (5095.ppm 
and 5118.ppm) as positive example, while an image of a bird on a tree under 
the sky is chosen as negative example (image 5521. ppm of Figure 8). Figure 9 
shows that images of birds are discarded (the noise reduced) and that more 
images of trees under the sky are retrieved (the miss decreased). 

Perfonmance evaluation 

[00104] In order to validate the proposed relevance feedback 

technique, we performed a perfomiance evaluation of our retrieval system. The 
evaluation was based on comparison between the use of positive example only 
and the use of both positive and negative examples. To perform any evaluation 
in the context of image retrieval, two main issues emerge: the acquisition of 
ground truth and the definition of performance criteria. For ground truth, we use 
human subjects: three persons participated in the all the experiences that we 
will describe below. Concerning performance criteria, the most used are 
Precision Pr and Recall Re [14]. In their simplest definition, precision is the 
proportion of retrieved images that are relevant, i.e., number of retrieved 
images that are relevant on the number of all retrieved images; and recall is the 
proportion of relevant images that are retrieved, i.e., number of relevant images 
that are retrieved on the number of all relevant images in the database. Some 
authors drew up the precision-recall curve Pr=f(Re) [14]; however, it has been 
observed, that this measure is less meaningful in the context of image retrieval 
since recall is consistently low [26] [11], Furthermore, we think that it is often 
difficult to compute recall, especially when the size of the image database is 
big; because this requires to know, for each query, the number of relevant 
images in a the whole database. Another problem with recall, is that it depends 
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strongly on the choice of the number of images to return to the user. If the 
number of relevant images In the database is bigger than the number of images 
retumed to the user, then the recall will be penalized. We adopt a more 
expressive curve which is the precision-scope curve Pr=f(Sc) [11]. Scope Sc Is 
the number of images retumed to the user, and hence the curve Pr=f(Sc) 
depicts the precision for different values of the number of Images retumed to 
the user. 

[00105] We carried out two experiences, each of which tries to 

measure a given aspect of our model. The first experience aims to measure the 
improvement, with negative example, in the relevance of retrieved images. The 
second experience aims to measure the improvement, with negative example, 
in the number of Iterations needed to locate a given category of images. 

First experience 

[00106] As mentioned above, the goal of the first experience is 1o 

measure the contribution of negative example in the improvement of the 
relevance of retrieved images. Each human subject participating in the 
experience was asked to fbmnulate a query using only positive example and to 
give a goodness score to each retrieved image, then to refine the results using 
negative example and to give a goodness score to each retrieved image. The 
possible scores are 2 if the image is good, 1 if the image is acceptable, and 0 If 
the image is bad. Each subject repeated the experience five times by 
specifying a new query each time. We computed precision as follows: Pr = the 
sum of degrees of relevance for retrieved images / the number of retrieved 
images. Figure 10 compares the curves Pr=f(Sc) in the two cases: retrieval with 
positive example and refinement with negative example. 
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[00107] We find that in average, when we introduce negative 

example, the Improvement in precision is about 20 %. In fact, the improvement 
varies from one query to another, because it depends on other factors such as 
the choice of a meaningful negative example and the constitution of the 
database. If, for a given query, the database contains a little number of relevant 
images, most of which have been retrieved in the first step, then the 
Introduction of negative example or any other technique will not be able to bring 
any notable improvement. 

Second experience 

[00108] The second experience attempts to measure the 

improvement in the number of refinement iterations needed to locate a given 
category of images, as well as the role of negative example In resolving the 
page zero problem (finding a good image to initiate the retrieval). We showed 
to each of our human subjects a set of images that are relatively similar to each 
other with respect to the color. None of the showed images appear in the set of 
images the subjects can use to formulate the initial query. Each subject is 
asked to locate at least one of the showed images using only positive example, 
and to count the number of iterations; then to restart the experience but using 
both positive and negative examples, and to count the number of iterations. 
This experience was repeated four times and the results are given in Figure 1 1 . 
SI, S2 and S3 designate respectively the three human subjects who 
participated In the experiments. PE means positive example and NE means 
negative example. Each entry in the table gives the number of iteretions 
needed to locate the searched images. 

[00109] Our first observation is that when they used both positive and 

negative examples, the subjects succeeded in all the experiences; however, 
when they used only positive example, some of them failed in certain 
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experiences to locate any sought image. In Experience 2.2 and Experience 2.4, 
at least one subject was unable to locate any sought image using positive 
example only. This is because, in a given iteration, all the retrieved images fall 
into an undesired category, and the formulation of the next-iteration query using 
any of these images leads to retrieve images belonging to the same category. 
The user can loop indefinitely, but will not be able to escape this situation by 
using positive example only. The second observation is that the use of negative 
example reduces appreciably the number of iterations. If we compute the 
average number of iterations among the successful experiences (2.1 and 2.3), 
we find 5.83 when only positive example is used, and 2.33 when both positive 
and negative examples are used. This experience shows clearfy the role of 
negative example in mitigating the page zero problem, indeed, after having 
obtaining at least one of the sought images, the user can use it to formulate a 
new queiy, and hence to retrieve more sought images. 

[00110] When peribrming content-based image retrieval, it Is very 

important to take into account the user's needs and specificities, which can be 
identified via relevance feedback. However, the use of positive example only 
isn*t always sufficient to determine what the user is looking for. This can be 
seen especially when all the candidate images to participate in the query 
appear in an inappropriate context or contain, in addition to the features the 
user Is looking for, features or objects that the user doesnt want to retrieve. 
Motivated by the importance of negative example, in this paper we studied its 
relevance in content-based image retrieval. We gave details and justifications 
on how it can be combined with positive example to perfomi feature weighting, 
and then explained how this can be applied to the retrieval process. This led us 
to propose a new model for positive and negative feedback. We validated our 
model by testing it on a heterogeneous database and perfomning some 
performance evaluations. 
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[001 1 1] It is to be noted that the present model is not limited to image 

retrieval but can be adapted and applied to any retrieval process with relevance 
feedback. 

[00112] Although the present invention has been described 

hereinabove by way of preferred embodiments thereof, it can be modified, 
without departing from the spirit and nature of the subject invention. 
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